DOCUMENT RESUME 



ED 460 145 



TM 033 619 



AUTHOR 

TITLE 

PUB DATE 
NOTE 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 

IDENTIFIERS 



Stewart, Robert G. 

Using Resampling Methods To Improve the Results of 
Meta~Analysis . 

2001-11-15 

lOp.; Paper presented at the Annual Meeting of the Mid-South 
Educational Research Association (30th, Little Rock, AR, 
November 14-16, 2001). 

Reports - Descriptive (141) -- Speeches/Meeting Papers (150) 
MFOl/PCOl Plus Postage. 

♦Educational Research; *Meta Analysis; Research Methodology; 
Sampling 

♦Resampling Techniques 



ABSTRACT 



As meta-analytic studies have become essential in 
educational research, resampling methods have matured and are supported by an 
in-depth theoretical and applied literature base. The objective of this paper 
is to assist researchers in applying resampling methods to the results of 
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Abstract 

Today, educational researchers must engage decades of primary research that is often 
interdisciplinary and from fields that continue to subspecialize. Consequently, meta-analytic 
studies are essential for cumulating findings and revealing new research opportunities. Indeed, as 
of June 30, 2001, the ERIC (Educational Research Information Center) database contained 1,516 
documents matching the keyword meta-analysis. Congruently, resampling methods have 
matured and are supported by an in-depth theoretical and applied literature base. Herein, the 
objective is to assist researchers with applying resampling methods to the results of meta- 
analysis. Accordingly, the discussion proceeds along three lines. First, the rationale for 
resampling meta-analyses is reviewed. Second, resampling applications and computational 
options are discussed. (Note that three SAS macro programs for resampling meta-analyses are 
presented.) Finally, to further explore the potential of resampling meta-analyses, a research 
agenda is proposed. 
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Using Resampling Methods to Improve the Results of Meta- Analysis 

In their watershed article, Adams, Gurevitch, and Rosenberg (1997) rhetorically ask “is . . 
. [the resampling of meta-analysis] really necessary ” (p. 1281). Indeed, such a question seems 
somewhat trite until one considers the computational intensity required of most resampling 
methods (e.g., bootstrapping [i.e., taking a random sample of n items with replacement from a 
group of size n repeated k times] and randomization tests). Accordingly, the decision to 
resample should involve an assessment of distributional assumptions that underlie a given meta- 
analysis (Adams, Gurevitch, & Rosenberg, 1997). If one senses that the assumptions are not 
being meet, then nonparametric resampling may yield more power than either parametric or 
nonparametric ranking approaches (Adams, Gurevitch, & Rosenberg, 1997). Indeed, for those 
with a penchant for analysis, using all applicable approaches would seem most reasonable. 

Resampling Applications 

Within the context of treatment effect meta-analysis, three resampling applications have 
been advanced. These are (a) testing for heterogeneity of effect size variance, (b) computing 
effect size confidence limits, and (c) testing for moderator variables. A discussion of each 
application now follows: 

Testing Variance Heterogeneity 

According to Wang and Bushman (1999), “effect-size estimates should not be combined 
unless they are homogeneous or similar in magnitude” (p. 19). Consequently, formal test for 
assessing homogeneity in meta-analyses have been developed (e.g., the Q statistic for within and 
between studies variation; Wang and Bushman, 1999). With regard to resampling, between- 
studies variation is of interest. The results from an initial application by Adams, Gurevitch, and 
Rosenberg (1997) indicates general agreement between resampling and parametric approaches. 
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A detailed explanation of the randomization procedure is provided by Adams, Gurevitch, and 
Rosenberg (1997, p. 1279). 

Computing Confidence Limits 

An essential aspect of meta-analysis involves the computation of effect size confidence 
limits (intervals). Accordingly, Adams, Gurevitch, and Rosenberg (1997, p. 1279) computed 
confidence limits using both percentile and bias corrected percentile methods. In general 
resampling confidence limits tended to be wider than parametric limits. In the following 
statement Adams, Gurevitch, and Rosenberg (1997 ) describe an interesting outcome: 

In one instance, primary producers, the effect size was considered to be different from 
zero using standard confidence limits, but not different from zero using bootstrap 
confidence limits, implying that, in some instances, confidence limits derived from 
standard methods may be less conservative in establishing the significance of small to 
moderate effects, (p. 1280) 

A detailed explanation of the bootstrap procedure is provided by Adams, Gurevitch, and 
Rosenberg (1997, p. 1279). 

Testing Moderator Variables 

In general, moderator variables interfere with the relationship between dependent and 
independent variables thereby causing a change in strength and/or direction (Baron & Kenny, 
1986, as cited in Wang & Bushman, 1999). In meta-analysis, “moderators are any known study 
characteristics [e.g., date of publication, type of research design] that are associated with 
differences in effect-size estimates between studies” (Wang & Bushman, 1999, p. 14). Although 
much debated has transpired regarding the best approach for assessing moderator effects in meta- 
analysis (see e.g.. Hall & Rosenthal, 1991; Viswesvaran & Sanchez, 1998), a consensus has not 
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been reached (Brown, Homer, & Inman, 1998). 

According to Brown, Homer, and Inman (1998) the bootstrap multivariate regression 
offers several advantages over existing approaches for moderator analysis (e.g., independent t- 
tests). These are (a) increased detection of statistically significant moderator effects and (b) 
simultaneous analysis of study characteristics as opposed to sequential (Brown, Homer, and 
Inman, 1998). A detailed explanation of the bootstrap procedure is provided by Brown, Homer, 
and Inman (1998, p. 120). 

Computational Options 

Most general purpose computing packages can perform meta-analyses (see Wang & 
Bushman, 1999, for examples using SAS). Moreover, many packages have been designed 
specifically for meta-analysis (see Normand, 1995, for a review). Based on the author’s 
collective review, only 1 package (i.e., MetaWin 1.0 and 2.0) supports the resampling of meta- 
analyses (see Rosenberg, Adams, & Gurevitch, 2000b, p. 46). It would be desirable to have a 
second option if for not other reason than to check one’s MetaWin output. Accordingly, three 
SAS macro programs have been developed by the author to perform the resampling analyses 
discussed herein (SAS Institute, 1990). These are (a) %QB_STAR, (b) %CI_STAR, and (c) 
%MA_STAR. It should be noted that the above programs require four SAS utility macros (a) 
%SEEDS, (b) %LEVELS, (c) %WITHIN, and (d) %RANDOM. With the exception of 
%WITHIN (Wang & Bushman, 1999, pp. 266-267), all programs will be made available upon 
request (see Author Note for contact information). 
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Research Agenda 

To improve the resampling of meta-analyses the following research agenda is proposed: 

1. Explore the possibilities of using resampling with other meta-analytic methods. For example 
(a) validity generalization, (b) reliability generalization, (c) factor structure generalization, (d) 
the synthesis of simulation results, and (e) multivariate effect sizes. (Note that the later is of 
current interest to the author.) 

2. Incorporate (or develop) resampling diagnostics for use with meta-analytic results. For 
example, (a) the iterative (or nested) bootstrap, and (b) the jackknife after bootstrap plot. 

3. Explore the use of sensitivity analysis in conjunction with the resampling of meta-analytic 
results. 

4. Assess the merits of existing meta-analysis programs for resampling. Questions of interests 
are (a) Which resampling features are common (or unique) among programs? (b) Which 
programs are easy (or difficult) to use? and (c) What discrepancies (if any) exist among program 
outputs? 

Conclusions 

Today, educational researchers must engage decades of primary research that is often 
interdisciplinary and from fields that continue to sub specialize. Consequently, meta-analytic 
studies are essential for cumulating findings and revealing new research opportunities. 
Resampling methods can improve these efforts. 
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